<html>
<head><meta charset="utf-8"><title>rust-analyzer#1213 · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html">rust-analyzer#1213</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="164388099"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388099" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388099">(Apr 28 2019 at 13:47)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> I've looked more at 1213 and I must say I don't entirely like the bifurcation behind compound and atomic tokens</p>



<a name="164388103"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388103" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388103">(Apr 28 2019 at 13:47)</a>:</h4>
<p>I wonder if we, with a more thorough refactoring, can make a better API?</p>



<a name="164388153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388153">(Apr 28 2019 at 13:48)</a>:</h4>
<p>let me scketch what I have in mind...</p>



<a name="164388176"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388176" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388176">(Apr 28 2019 at 13:48)</a>:</h4>
<p>What we do it is in  the <code>Parser</code> level</p>



<a name="164388276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388276">(Apr 28 2019 at 13:50)</a>:</h4>
<div class="codehilite"><pre><span></span>    #[inline(always)] // to make sure compund case is compiled away when `kind` is compile time constant
    pub(crate) fn at(&amp;self, kind: SyntaxKind) -&gt; bool {
        match kind {
            DOTDOT =&gt; self.at2(DOT, DOT),
            DOTDOTDOT =&gt; ...
            _ =&gt; self.current() == kind
        }
    }

    // Private impl details
    fn at2(&amp;self, kind1: SyntaxKind, kind2: SyntaxKind) -&gt; bool {
        self.current2() == Some((kind1, kind2))
    }
</pre></div>



<a name="164388286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388286">(Apr 28 2019 at 13:51)</a>:</h4>
<p>Could we do something like this?</p>



<a name="164388301"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388301" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388301">(Apr 28 2019 at 13:51)</a>:</h4>
<p>That is, the grammar impl will just ask <code>if at(DOTDOT)</code>, without thinking too much if it is a compound token</p>



<a name="164388370"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388370" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388370">(Apr 28 2019 at 13:52)</a>:</h4>
<p>And the same trick could be applied to <code>at_ts</code>: we include multi-char tokens in tokensets, and internally we run special logic for multibyte bits of ts</p>



<a name="164388381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388381">(Apr 28 2019 at 13:53)</a>:</h4>
<p>Yes of course, but it is not so efficient if we concern it,  as we have to check it every time. If it is not an concern, we can do it too. But we have to do it in all at, nth, eat methods</p>



<a name="164388386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388386">(Apr 28 2019 at 13:53)</a>:</h4>
<p>I <em>think</em> it could be just as efficient</p>



<a name="164388389"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388389" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388389">(Apr 28 2019 at 13:53)</a>:</h4>
<p>the trick is that <code>kind</code> is almost always a compile-time constant</p>



<a name="164388436"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388436" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388436">(Apr 28 2019 at 13:54)</a>:</h4>
<p>So, if llvm inlines stuff, it will compile away the <code>match kind</code> branch</p>



<a name="164388461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388461">(Apr 28 2019 at 13:55)</a>:</h4>
<p>The only place where abstraction will leak are <code>match p.current()</code> branches in parser</p>



<a name="164388504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388504">(Apr 28 2019 at 13:56)</a>:</h4>
<p>instead of </p>
<div class="codehilite"><pre><span></span>match p.current() {
    DOTDOT =&gt;
}
</pre></div>


<p>You'll have to write</p>
<div class="codehilite"><pre><span></span>match p.current() {
    DOT if p.at(DOTDOT) =&gt;
}
</pre></div>


<p>But that presumably is ok?</p>



<a name="164388535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388535">(Apr 28 2019 at 13:57)</a>:</h4>
<p>That's okay.</p>



<a name="164388591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388591">(Apr 28 2019 at 13:58)</a>:</h4>
<p>Let me try your method first, and let see if there is any edge case :)</p>



<a name="164388611"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388611" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388611">(Apr 28 2019 at 13:59)</a>:</h4>
<p>Yeah, there could be edge cases especially around lookahead, but maybe lookahead can be restricted to atomic tokens?</p>



<a name="164388682"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164388682" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164388682">(Apr 28 2019 at 14:00)</a>:</h4>
<p>To check performance, one can use <code>cargo run --release -p ra_cli -- parse --no-dump  &lt; some-big-file.rs</code>. A good candidate for <code>some-big.file.rs</code> is te parser from rust-lang/rust repo</p>



<a name="164394291"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%231213/near/164394291" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.231213.html#164394291">(Apr 28 2019 at 16:02)</a>:</h4>
<blockquote>
<p>but maybe lookahead can be restricted to atomic tokens</p>
</blockquote>
<p>It seem to be not work, but i did manage to handle it in <code>Parser::nth</code> , I found that we already have the <code>dollar</code> handling code in it, so ... maybe we should not be too worry about performance right now <span aria-label="rolling on the floor laughing" class="emoji emoji-1f923" role="img" title="rolling on the floor laughing">:rolling_on_the_floor_laughing:</span></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>